feat(outbound): close the provider seam for every sender (B7) - #1000
Merged
Conversation
…asons Two additive local failure reasons for the sending-protection holds: submission.policy_budget_expired (a sending-budget hold reached its seven-day deadline) and submission.sending_setup_expired (SES tenant readiness did not land within the 72-hour setup deadline). Both are local, correctable outcomes like submission.local_retries_exhausted, and neither may ever be reported as a recipient rejection or a provider outage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
… paths Two callers hold provider evidence but no token: the worker that finds provider-accept evidence already recorded on a row it is about to re-drive, and the terminal reconciler settling a stranded row from that same evidence. Neither can name an ordinal. SettleOperation applies the outcome to the latest attempt whose provider call started — never a later ordinal that was only reserved, and nothing when no attempt ever dialed — through the same body SettleProvider uses. LookupOperation recovers a reference for an operation that already exists; it is not a constructor, and every Gate method still reloads the row under lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The send claim now returns the message's persisted hold class and anchor (migration 116) plus the owning account's last_resumed_at and ses_tenant_ready_at, so every worker execution can re-derive the same deadline. RecordOutboundHold writes the pair only while the message is pre-terminal; every terminal write — sent, failed, evidence-settled, trash-cancelled — clears it, so a stale hold can never outlive its message's outcome. The two new local expiry reasons are recognized as complete terminal fallbacks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The outbound send worker now authorizes every provider call through the sending-protection Gate, in the fixed order the design names: Reserve the durable attempt; snooze on an early hold without provider I/O; DeferAttempt on a rate deferral and CancelAttempt on a final suppression match; ConsumeAttempt as the last serialized decision; then the authorized submitter, which redeems the token immediately before the socket opens and settles the provider's answer. A later execution after a confirmed attempt returns to Reserve, which allocates the next ordinal. The worker-owned RampGate and agent.NewOutboundRampGate are removed: the ramp is composed inside the gate and its progress moves only through settlement. The Deliverer contract carries the token; the production deliverer is outbound.ProviderSubmitter and refuses to dial without one. A lost 250 (ErrProviderAcceptanceUnknown) is retried as a new ordinal and never settled. Enqueue prepares the operation in the accept transaction, between the message insert and the River insert; a paused account is refused there (ErrSendingPaused, HTTP 403 sending_paused) rather than queued. Jobs from a pre-floor slot carry no reference and resolve at fire time through the same Prepare path. Finite holds persist a class and anchor on the message and derive the deadline every execution: 72 hours for rate/ramp/provider and tenant setup, seven days for policy budget. The first finite hold anchors at the latest of accept, schedule, review, and last resume; a budget hold promotes any class and keeps the anchor; policy_budget never changes again; tenant readiness landing inside the setup deadline moves the class to rate/ramp/provider exactly once; a pause has no clock but a running deadline keeps running. Expiry emits the class's own reason. Terminal reconciliation is settlement-only: an evidence-settled row also settles the attempt that dialed through Gate.SettleOperation. cmd/e2a gains one composition root (newOutboundSending) and a wiring test that proves the registered send path holds the concrete gate and the ProviderSubmitter-backed deliverer. The test servers build the same composition with the disabled policy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The machine-checked contracts caught three vocabularies the worker cutover widened without saying so: the error-code catalog and the ErrorBody.Code documentation (sending_paused, 403, auth family), the lifecycle reason table in docs/api.md (submission.policy_budget_expired, submission.sending_setup_expired), and the OpenAPI description the two generated SDK models embed. Both SDK error maps classify sending_paused as a non-retryable permission error, with tests. The email-eval integration runner's job-args parser insisted on exactly one key; the accept transaction now stamps operation_ref beside message_id, so the parser admits that key and still rejects any other. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Two parallel reviews (correctness + adversarial) over the first cut. Every item has a named test. Contract surfaces (the blocker in both): the two new lifecycle reasons join the hand-maintained reason_code enum tag, both closed-vocabulary tests, the regenerated spec, both generated SDK models (mirrored by hand; the generator needs Docker, and the description now carries no apostrophe so the two generators agree), the web lifecycle parser and timeline, docs/api.md and docs/events.md. sending_paused is registered in the error catalog, the ErrorBody.Code doc, docs/api.md, and both SDK error maps. Worker: - A gate outage is a bounded rate/ramp/provider hold, not an unbounded snooze. - A paused job evaluates no deadline; a persisted deadline is not extended and the first hold after resume applies it. - A provider outage never emits the setup reason (expiryReasonFor). - markFailed's evidence-settle branch settles the dialed attempt, as the reconciler already did; a failed post-acceptance settlement is retried before it is logged as critical; a provider-id conflict is surfaced as an invariant alarm. - The job's operation reference must name its own message; a mismatch cancels before any ledger call. Enqueue refuses a zero reference. - HoldClassFor maps reasons by name; the armed worker RegisterJobs builds is exposed (Jobs.SendWorker) so the wiring test can prove it carries the gate and the legacy resolver and the submitter carries the configuration set. Gate: SettleOperation prefers the oldest dialed attempt with no provider id yet, so evidence arriving in send order binds each attempt when several dialed. Paused accounts on every enqueue path: 403 sending_paused on the direct, platform-test, and HITL-approve paths; the TTL auto-approve sweep defers a paused account's expired review by an hour (DeferReviewExpiry) instead of re-picking it first every cycle and starving the batch. The email-eval integration runner admits the operation_ref args key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Mutation-tested re-review of the previous fix round: no blockers, six should-fixes. - SettleOperation without a token resolves the attempt as: one already bound to this exact provider id (a replay stays home), else the oldest dialed attempt with no id, else the latest dialed. The earlier oldest-unbound-first rule let a replay for attempt one bind attempt two; the test now covers that shape. - snoozeOnGateError threads the live reservation into the bounded hold, so an expiry at final authorization gives the attempt back instead of stranding it under an enforcing policy. - MarkFailed returns the evidence's provider id, and the worker's evidence settle under a terminal write carries it — the reconciler already did. The two evidence paths now agree. - resettle logs at critical level when the context ends mid-retry; a dedicated test covers the retry itself. - The wiring test registers workers exactly as main does and inspects the worker River received (Jobs.RegisteredSendWorker), so a RegisterJobs that bypassed the armed constructor fails it. - sending_paused is marked experimental beside blocked_by_policy in the stability extension, the docs, and the description, since the pause control ships disabled and pre-GA. The Python forward-compat table gains the two lifecycle reasons. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Slice B7 of the sending abuse prevention plan. The relay no longer exports a send method; ProviderSubmitter.SubmitOnce with a gate token is the only way to reach the provider, and a tracked-closure test parses every production file to keep it that way. - hitlnotify + webhooknotify: enqueue prepares a customer_notification operation in the source transaction and stamps it on the job; workers run Reserve -> early hold -> ConsumeAttempt -> authorized submit and snooze on a hold without provider I/O; pre-floor jobs resolve at fire time and are stamped once (jobs.StampJobArg). - public feedback: server-keyed public_feedback_notification operation with a bounded per-attempt Reserve/Consume/Submit loop; a definite rejection or a lost acceptance stops it. - e2a -reconcile-legacy-sending-jobs: stamps pending outbound_send / hitl_notify / webhook_notify jobs that carry no operation, cancels orphans, exits nonzero unless every scanned job was decided. - main, TestServer and the contract server build the notifiers over the shared submitter and hand the API the submitter + gate. - design addendum in docs/design/async-message-pipeline.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
…o feat/sending-provider-closure
Review round 1 of the provider-seam closure (B7). - PrepareNotificationTx derives the operation id from its source (op_hitl_<message>, op_wh_<kind>_<webhook>_<episode>), so a repeat preparation yields one operation and the notify workers cancel a job whose reference names any other operation. - The notify Deliverer is Compose + Submit; workers run compose -> Reserve -> hold -> ConsumeAttempt -> Submit, so a compose failure charges nothing and the token is consumed right before the socket. - Reconcile command re-reads each job under FOR UPDATE and skips one a worker claimed or stamped meanwhile; counts separate paused/skipped. - Feedback loop submits the token's canonical recipients, paces retries to fit the handler budget, keeps the SMTP error on a deadline, releases a reserved attempt when authorize errors, no panic on id mint. - Closure guard never skips, matches method references, exempts the SubmitOnce symbol only, asserts its sentinel, fences the SES v2 import. - Webhook health notices older than seven days are dropped. - Wiring test for the notification bundles and the API seam; StampJobArg tests for the jobs coverage floor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Re-review of the provider-seam closure (B7) found that migration 113 stamped adopted notify jobs with op_<md5> references, which the new source binding would have cancelled on any upgrade crossing v1.8.7. - A reference that is not a derived id is treated as pre-derivation: the notify workers re-resolve it through the Prepare path and replace it once (jobs.SetJobArg); a derived id for another source still cancels. - The reconcile command scans and re-keys those references too. - Bounded the feedback attempt release (2s); symmetric 7-day age guard on HITL notices; episode key in microseconds; nil-receiver guards on the compose path; doc and comment corrections; the closure guard states its residual scope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Round-3 re-review nits: COALESCE the conforming-reference predicate so a reference with no id cannot fall out of a NOT scan, decode only the source fields so such a reference is replaced rather than failing to decode, and state in the workers that any non-derived shape re-derives from the job's own source. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
…-closure # Conflicts: # cmd/e2a/outbound_wiring.go # cmd/e2a/sending_policy_wiring_test.go # docs/design/async-message-pipeline.md # internal/testutil/contract_server.go # internal/testutil/server.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice B7 of the sending abuse prevention plan (ops PR #320): close the provider seam B5 opened. Stacked on #999 (B6); base is
feat/sending-worker-cutoverand will be retargeted tomainonce B6 merges.What changes
The relay can no longer send without a token.
outbound.SMTPRelay's exportedSend*methods andoutbound.Sender'sSend/SendOnce/SubmitOnce*are deleted. The only socket-opening entry point isProviderSubmitter.SubmitOnce(ctx, auth, envelope). A new tracked-closure test (internal/outbound/provider_authorization_guard_test.go) parses every production Go file and fails on anynet/smtpimport or any call to the relay's socket core outside the named exceptions, so a future bypass cannot land silently.Notification paths cross the gate.
hitlnotify(approval emails) andwebhooknotify(webhook health notices):customer_notificationoperation in the same transaction as the source row (PrepareNotificationTx) and stamps it on the job (operation_ref);jobs.StampJobArg, key-absent guard, existing fields preserved).Public feedback mail (
POST /api/feedback) is apublic_feedback_notificationoperation keyed by a server-minted submission id; its envelope is the configured notify set, never the request. The in-request retry loop is bounded (4 attempts), every physical attempt is its own charged ordinal with its own attempt id on the wire, and a definite rejection or a lost acceptance stops the loop.Operator command
e2a -reconcile-legacy-sending-jobs: stamps an operation onto every pendingoutbound_send/hitl_notify/webhook_notifyjob that has none (through the exact Prepare path its enqueue would have used), cancels orphans whose source row is gone, leaves paused-account jobs to the worker's hold path, prints counts, and exits nonzero unless every scanned job was decided. Each job is one transaction; a failure leaves it untouched for a rerun.Wiring:
cmd/e2a/main.go,testutil.TestServer, and the contract server build the notifiers over the sharedProviderSubmitterand hand the API the submitter + gate (SetProviderSubmitter).Review round 1 (correctness + adversarial, Opus) → fixes in the second commit
op_hitl_<msg>,op_wh_<kind>_<webhook>_<episode>):PrepareNotificationTxis idempotent per hold / per health episode, and both notify workers cancel a job whose reference names any other operation — the binding the B6 message worker has.Delivereris nowCompose+Submit; the notify workers run compose → Reserve → hold → ConsumeAttempt → Submit, so an owner-lookup / signing / DKIM failure costs no ordinal and the token is consumed immediately before the socket. The startup "notifier not wired yet" retry now also costs nothing.FOR UPDATEinside its own transaction and skips one a worker claimed or stamped meanwhile; output separatespaused/skippedfromremaining.go.mod, git optional), matches method values / method expressions, exception is theSubmitOncesymbol not the file, asserts its sentinel is live, and fences the SES v2 SDK import to sender-identity provisioning.StampJobArgtests for theinternal/jobscoverage floor.Review round 2 (mutation-tested re-review, Opus) → third commit
Every round-1 fix held under mutation. One blocker found in the fix commit itself: migration 113 (v1.8.7) stamped adopted notify jobs with
op_<md5>references, which the new binding would have cancelled on any upgrade crossing v1.8.7. Now a reference that is not a derived id is treated as pre-derivation: the worker re-resolves it through the Prepare path and replaces it once (jobs.SetJobArg); the reconcile command scans and re-keys those too. Also: bounded the feedback attempt release (2s), symmetric 7-day age guard on HITL notices, episode key in microseconds, nil-receiver guards on the new compose path, doc/comment corrections, and the guard's residual scope stated in its comment.Design-level, not changed here (needs a decision): public feedback charges the two global customer pools by spec (design §"public feedback consumes both global pools", A4 gate). The adversarial reviewer measured 8 probation units per request (4 attempts × 2 recipients) behind the 10/hr/IP limit, i.e. one IP can drain the 150/day probation pool in under 8 hours once
budget_mode: enforceis armed. Options: a dedicated small feedback pool, a tighter per-IP limit, or a daily platform cap on feedback mail. Dormant while the policy is disabled.Behaviour note: notification and feedback mail now carries
X-SES-CONFIGURATION-SET, so SES publishes delivery feedback for it; it correlates to no message row and the SNS consumer acks it as unknown (a log line, no suppression).Not in this PR
Tests
internal/outbound: closure guard; relay tests over the unexported core.internal/hitlnotify,internal/webhooknotify: gated-order worker tests (authorize-then-deliver, early/late hold and gate error snooze without I/O, legacy resolve+stamp once), notifier tests through the submitter, e2e harness over the gate.internal/agent: feedback seam tests with a scripted SMTP server (retry is a new ordinal, 5xx not retried, lost acceptance not retried, bounded retries, envelope is configuration; wire carries the attempt header).cmd/e2a: reconcile command (stamp live, cancel orphan, skip finalized and non-submitting kinds, stamped ref round-trips, second pass empty; undecided job reported and untouched, nonzero exit).Local: fmt/vet clean; affected suites and
-raceon the three notify/outbound packages green.internal/e2ehas known environmental failures on cleanmainlocally; CI is the authority.🤖 Generated with Claude Code
https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX